Visual analysis of viseme dynamics

Face to face dialogue is the most natural mode of communication between humans. The combination of human visual perception of expression and perception in changes in intonation provides semantic information that communicates idea, feelings and concepts. The realistic modelling of speech movements, through automatic facial animation, and maintaining audio-visual coherence is still a challenge in both the computer graphics and film industry. A common approach to producing visual speech is to interpolate parameters that describe mouth variation in sequence, known as visemes. A viseme corresponds to a phoneme in an utterance. Most talking head systems use sets of static visemes, represented by a single mouth shape image or 3D model. However, discretising visemes in this way does not account for context-dependent dynamic information, coarticulation. This thesis presents several visual analysis and dynamic modelling techniques for visual phones. This spans several areas of work, from capture and representation through to analysis and synthesis of speech movements and coarticulation. A novel method is reported for the automatic extraction of inner-lip contour edges from sequences of mouth images in speech. The proposed detection technique is a key-frame exemplar-based method that is not dependent on any prior frame information for intitialisation allowing for reliable and accurate inner-lip localisation for large frame to frame changes in lip-shape inherent in 25Hz video of visual speech. Visual analysis of phonemes in continuous speech is performed, that involves the investigation of mouth representations as well as a comparative analysis between static and dynamic representations of visemes. The analysis shows the need to analyse and model the underlying dynamics of visemes due to coarticulation. Finally, visual analysis of lip coarticulation in Vowel-Consonant-Vowel (VCV) utterances is presented. Based on ensemble statistics a novel approach to analysis and modelling of temporal dynamics is presented. Results show that the temporal influence of coarticulation is significant both in lip shape variation and timings of lip movement during coarticulation. This work shows that the effect of temporal variation due to coarticulation is statistically significant and should be taken into account in modelling visual speech synthesis. The work in this thesis provides the foundation for further research towards achieving perceptually realistic animation of a talking head and the understanding of visual dynamics of shape and texture during speech.